Problem

HdfsReader.selectFieldsFromGenericRecord serializes Avro string fields as JSON string primitives via JsonObject.addProperty, which double-encodes pre-serialized JSON payloads when the output is used as an outbound HTTP request body. The ARRAY and RECORD branches already inline via gson.fromJson(...) + jsonObject.add(...); the STRING/UNION branch has no equivalent.

Use case

JSON-LD payloads use @context / @type field names, which are not valid Avro identifiers, so a structured Avro record cannot represent them. The workaround is to store a pre-serialized JSON-LD document as a string field in Avro, but DIL currently escapes such strings when emitting to an HTTP POST body, producing malformed payloads.

Change

Adds a new boolean property ms.hdfs.reader.parse.json.strings (default false). When enabled, the STRING/UNION branch of selectFieldsFromGenericRecord attempts to parse string values that look like JSON ({…} or […]) via gson.fromJson(..., JsonElement.class) and inlines the result as a JsonElement. Parse failures fall back to the existing string-primitive behavior.

The implementation matches the style of the adjacent ARRAY and RECORD branches, which already use gson.fromJson(..., JsonArray.class) and gson.fromJson(..., JsonObject.class). Using the polymorphic JsonElement.class here lets the new branch handle both object-rooted and array-rooted JSON content without pre-classifying the input.

Back-compatibility

• Property default is false; existing jobs are unaffected.
• New code path has a try/catch fallback to the existing behavior on parse failure.
• No new dependencies (Gson is already on the classpath).

Tests

Added 8 TestNG cases in HdfsReaderTest (cdi-core/src/test/java/com/linkedin/cdi/util/HdfsReaderTest.java) using Whitebox.invokeMethod for the private method, matching the pattern in AvroExtractorTest:

Flag on:

• JSON object → inlined as JsonObject
• JSON array (JSON-LD shape with @context) → inlined as JsonArray
• Plain text → preserved as JsonPrimitive
• Malformed JSON (e.g. [{incomplete]) → falls back to JsonPrimitive
• null value → JsonNull
• Empty string → preserved as empty JsonPrimitive

Flag off (back-compat gate):

• JSON-looking content stays as a JsonPrimitive (the regression the flag must not cause)
• Plain text unchanged

All 8 tests pass locally on JDK 1.8 + Gradle 6.8.1 (matches CI environment).

Files changed

• cdi-core/src/main/java/com/linkedin/cdi/configuration/PropertyCollection.java — register MSTAGE_HDFS_READER_PARSE_JSON_STRINGS
• cdi-core/src/main/java/com/linkedin/cdi/util/HdfsReader.java — conditional inlining branch + looksLikeJson helper
• cdi-core/src/test/java/com/linkedin/cdi/util/HdfsReaderTest.java — new test class
• docs/parameters/ms.hdfs.reader.parse.json.strings.md — new property doc page

Documentation

New page at docs/parameters/ms.hdfs.reader.parse.json.strings.md describing behavior, default, and an example contrasting flag-off vs flag-on output.

E2E test validation

PR in guest-workflows-spark for snapshot test, where we pointed to the snapshot version of dil-internal and passed the new flag as true

https://github.com/linkedin-multiproduct/guest-workflows-spark/pull/362

verified the DAG ran successfully in guest-workflows-spark with the new flag passed as true

confirmed from Google POC they are able to see JSON object from snapshot test